---
title: "Replication Materials"
subtitle: "The Causal Effects of Political Incivility in Social Media Discussions"
author:
  - Maxwell B. Allamong (University of Houston)
  - Chris Bail (Duke University)
  - D. Sunshine Hillygus (Duke University)
  - Alexander Volfovsky (Duke University)
date: "2026-03-23"
output:
  html_document:
    toc: true
    toc_depth: 3
    toc_float: true
    number_sections: true
    theme: flatly
    highlight: tango
  pdf_document:
    toc: true
    toc_depth: 3
    number_sections: true
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE, eval = FALSE)
```

# Overview

This replication package contains the scripts and data needed to replicate all analyses in "The Causal Effects of Political Incivility in Social Media Discussions."

The paper reports an experiment in which participants were recruited via Prolific and randomly assigned to a civil or uncivil condition on **Spark Social**, a purpose-built social media application. Participants in the uncivil condition were exposed to a newsfeed populated with uncivil pre-populated posts and synthetic user (bot) comments; participants in the civil condition saw civil equivalents. The treatment period lasted approximately 12 minutes. Pre- and post-treatment surveys measured political attitudes including affective polarization, climate policy opinion polarization, political trust, and satisfaction with democracy. Platform data capture behavioral outcomes including the toxicity, volume, and content of participant-generated posts and comments.

The materials include:

- A copy of the manuscript (`Causal-Effects-of-Political-Incivility.pdf`)
- Survey data from eligibility screening, pre-treatment, and post-treatment surveys
- Platform data from the Spark Social newsfeed (participant posts, participant comments, and bot comments)
- Reference and pretest data used to construct and validate the treatment stimuli
- The 2024 ANES Time Series Study (for sample benchmarking)

# Computing Environment

| Component | Details |
|:----------|:--------|
| Operating System | macOS Tahoe 26.3.1 |
| Processor | Apple M2 |
| Memory (RAM) | 8 GB |
| System Type | 64-bit operating system, ARM64-based processor |
| R Version | 4.5.0 (2025-04-11) |
| RStudio Version | 2026.1.1.403 |
| **Estimated Runtime** | **~2 minutes** |

# Quick Start

## Step 1: Download and Unzip
Download and unzip the replication materials to your desired location.

## Step 2: Open the Project

Open `Incivility.Rproj` in RStudio (or set your working directory to the root folder).

## Step 3: Install Required Packages

Install the required R packages. An `install.packages()` command has been included at the beginning of the `Incivility-Analysis.R` script to perform the installation. 

```{r install-packages}
install.packages(c("readxl", "tidyverse", "vtable", "lmerTest", "cowplot",
                   "here", "xtable", "RColorBrewer", "stargazer", "survey",
                   "srvyr"))
```


## Step 4: Run the Analysis

Run `Incivility-Analysis.R` the entire script from beginning to end.

## Step 5: Find Outputs

Outputs will be saved to:

- **Figures:** PNG files in `Figures/` folder
- **Tables:** LaTeX files in `Tables/` folder
- **Log:** `Incivility-Analysis-log.txt` in `Logs/` folder

# Scripts

## Incivility-Analysis.R

**Description:** Main analysis script that produces all tables and figures from the main text and Supplementary Information except for the following:

- Figure 1: Screenshots from the Spark Social mobile application
- Figure 2: Examples of content from uncivil and civil newsfeeds
- Figure S1: Flowchart of the study procedure
- Figure S8: Flowchart of attrition (values produced by script, but not figure)
- Table S1: Schematic of Spark Social newsfeed

# Codebooks

Variable-level documentation for all datasets is in the `Codebooks/` folder.

| File | Description |
|:-----|:------------|
| `Incivility-Codebooks.Rmd` | Variable dictionaries for all study datasets (see below for dataset inventory) |
| `anes_timeseries_2024_userguidecodebook_20250430.pdf` | Official ANES 2024 Time Series Study user guide and codebook |

> **Note:** The ANES dataset (`anes_timeseries_2024_csv_20250430.csv`) is documented in its official codebook above. All other datasets are documented in `Incivility-Codebooks.Rmd`.

# Data Files

All data files are located in the `Data/` folder with the following structure:

```
Data/
  |-- Survey Data/
  |-- Platform Data/
  |-- Reference Data/
  |-- Pretest Data/
```

## Survey Data

Located in `Data/Survey Data/`

| # | File | Description |
|:--|:-----|:------------|
| 1 | `civility-full-ajps.xlsx` | Full survey data with summarized platform variables (main analysis dataset) |
| 2 | `civility-intake-attrition.xlsx` | Intake/eligibility survey data for attrition analysis |
| 3 | `civility-pre-attrition.xlsx` | Pre-treatment survey data for attrition analysis |
| 4 | `civility-post-attrition.xlsx` | Post-treatment survey data for attrition analysis |
| 5 | `anes_timeseries_2024_csv_20250430.csv` | 2024 ANES Time Series Study Preliminary Release |

**ANES Data Source:** American National Election Studies. 2025. ANES 2024 Time Series Study Preliminary Release. [dataset and documentation]. April 30, 2025 version. <https://electionstudies.org/data-center/2024-time-series-study/>

## Platform Data

Located in `Data/Platform Data/`

| # | File | Description |
|:--|:-----|:------------|
| 6 | `all-posts-2025-10-12.xlsx` | All participant-generated posts from Spark Social platform |
| 7 | `all-comments-2025-10-12.xlsx` | All participant-generated comments from Spark Social platform |
| 8 | `all-bot-comments-2025-10-12.xlsx` | All synthetic user (bot) comments from Spark Social platform |

## Reference Data

Located in `Data/Reference Data/`

| # | File | Description |
|:--|:-----|:------------|
| 9  | `posts.xlsx` | Researcher-generated posts from the civil and uncivil newsfeeds |
| 10 | `prolific-prepped.xlsx` | Anonymized Prolific IDs with completion code indicators |

## Pretest Data

Located in `Data/Pretest Data/`

### Text Measures (NLP analysis of pretested content)

| # | File | Description |
|:--|:-----|:------------|
| 11 | `TextMeasures-Posts.xlsx` | NLP measures for researcher-generated posts |
| 12 | `TextMeasures-Comments.xlsx` | NLP measures for researcher-generated comments |
| 13 | `TextMeasures-GPT-Comments.xlsx` | NLP measures for GPT-generated comments |

### Connect Pretest (Human ratings from Connect platform)

| # | File | Description |
|:--|:-----|:------------|
| 14 | `ConnectMeasures-Posts.xlsx` | Human civility ratings of researcher-generated posts |
| 15 | `ConnectMeasures-Comments.xlsx` | Human civility ratings of researcher-generated comments |
| 16 | `ConnectMeasures-GPT-Comments.xlsx` | Human civility ratings of GPT-generated comments |
| 17 | `ConnectMeasures-Posts-Pairs.xlsx` | Paired comparisons for researcher-generated posts |
| 18 | `ConnectMeasures-Comments-Pairs.xlsx` | Paired comparisons for researcher-generated comments |
| 19 | `ConnectMeasures-GPT-Comments-Pairs.xlsx` | Paired comparisons for GPT-generated comments |

# Outputs

## Figures

Located in `Figures/`

### Main Text Figures

| File | Description |
|:-----|:------------|
| `Figure3_coef_plot_h1_h4.png` | Coefficient plot for H1-H4 (main effects) |
| `Figure4_coef_plot_nlp.png` | Coefficient plot for NLP text measures |
| `Figure5_comment_behavior_by_post_type.png` | Comment behavior by post type |
| `Figure6_coef_plot_h5_h7.png` | Coefficient plot for H5-H7 |

### Supplementary Figures

| File | Description |
|:-----|:------------|
| `FigureS2_pretest_nlp.png` | NLP text features of pretested content |
| `FigureS3_pretest_human_mean_civility.png` | Mean civility ratings (Connect pretest) |
| `FigureS4_pretest_human_identifying_civility.png` | Correct identification of uncivil content |
| `FigureS5_bot_id_by_condition.png` | Bot identification by condition |
| `FigureS6_bot_comments_by_user_content.png` | Bot comments by user content type |
| `FigureS7a_post_distribution_by_condition.png` | Post distribution by condition |
| `FigureS7b_comment_distribution_by_condition.png` | Comment distribution by condition |
| `FigureS9_toxicity_by_post_type_by_condition.png` | Toxicity by post type |
| `FigureS10_comment_toxicity_by_time_by_condition.png` | Toxicity over time |

> **Note:** Figure S1 (flowchart) and Figure S8 (attrition) are not generated by the analysis script. Values for Figure S8 are computed and printed to the log file.

## Tables

Located in `Tables/`

### Supplementary Tables

| File | Description |
|:-----|:------------|
| `TableS2_desc_categorical.tex` | Summary statistics for categorical variables |
| `TableS3_desc_continuous.tex` | Summary statistics for continuous variables |
| `TableS4_desc_comparisons.tex` | Sample comparisons to ANES populations |
| `TableS5_desc_platform.tex` | Summary statistics for platform activity |
| `TableS6_balance.tex` | Balance table across conditions |
| `TableS7_toxicity_by_user_type.tex` | Toxicity comparisons by user type |
| `TableS8_ate_h1_h3.tex` | Average treatment effects (H1-H3) |
| `TableS9_ate_h4_h7.tex` | Average treatment effects (H4-H7) |
| `TableS10_ate_change.tex` | First-difference models |
| `TableS11_ate_toxicity_random_intercepts.tex` | Toxicity with random intercepts |
| `TableS12_ate_post_convo_features.tex` | Treatment effects on post features |
| `TableS13_ate_comment_convo_features.tex` | Treatment effects on comment features |
| `TableS14_ate_post_convo_features_random_intercepts.tex` | Post features with random intercepts |
| `TableS15_ate_comment_convo_features_random_intercepts.tex` | Comment features with random intercepts |
| `TableS16_means_diffs_convo_features.tex` | Mean differences in text features |
| `TableS17_comment_behavior_by_post_type.tex` | Commenting behavior by post type |

> **Note:** Table S1 (newsfeed schematic) is not generated by the analysis script.

## Log File

| File | Description |
|:-----|:------------|
| `Logs/Incivility-Analysis-log.txt` | Console output from full script execution, including all printed statistics, attrition summaries, and model outputs |

# R Packages

| Package | Version | Purpose |
|:--------|:--------|:--------|
| readxl | 1.4.5 | Reading Excel files |
| tidyverse | 2.0.0 | Data manipulation and visualization |
| vtable | 1.4.8 | Summary statistics tables |
| lmerTest | 3.1.3 | Linear mixed-effects models with tests |
| cowplot | 1.2.0 | Plot composition |
| here | 1.0.1 | Working directory management |
| xtable | 1.8.4 | LaTeX table generation |
| RColorBrewer | 1.1.3 | Color palettes for visualizations |
| stargazer | 5.2.3 | Regression table output |
| survey | 4.4.2 | Survey-weighted statistics |
| srvyr | 1.3.0 | Dplyr-style survey analysis |

# Citation

Please cite the replication materials as:

> Allamong, Maxwell B., Christopher A. Bail, D. Sunshine Hillygus, and Alexander Volfovsky. 2026. "Replication data for: The Causal Effects of Political Incivility in Social Media Discussions." *American Journal of Political Science Dataverse*. <https://doi.org/10.7910/DVN/L2ONWL>

# Contact

For questions about these materials, please contact:

**Max Allamong**
Email: [maxallamong@gmail.com](mailto:maxallamong@gmail.com)
